Can verbal autopsies be used on a national scale? Key findings and lessons from South Africa's national cause-of-death validation study

ABSTRACT Background Verbal autopsy (VA), though imperfect, serves as a vital tool to determine cause-of-death, particularly for out-of-facility deaths, but challenges persist in integrating VA into Civil Registration and Vital Statistics systems. Objective To describe the challenges and successes of collecting a national sample of verbal autopsy interviews in South Africa to obtain the cause of death profile in 2017/18. Methods We recruited next of kin from 27 randomly selected sub-districts (10.5%) across South Africa between September 2017 and April 2018. Trained fieldworkers conducted face-to-face interviews using the WHO2016 VA instrument, with physicians certifying underlying causes of death. Feasibility was evaluated based on response rates, participation, and data quality. Results Of the total 36,976 deaths registered, only 26% were identified during recruitment, with a 55% overall response rate for VA interviews. Physician-reviewed VA data were deemed of good quality for assigning underlying causes of death in 83% of cases. By comparing cause-specific mortality fractions, physician-reviewed VA identified 22.3% HIV/AIDS and InterVA-5 identified 18.5%, aligning with burden of disease estimates, while Statistics South Africa reported 4.9% HIV/AIDS. Conclusions The study demonstrated the feasibility of using VA on a national scale, but immense challenges in identifying and recruiting next of kin highlight the importance of formalising VAs within the country’s death notification system.

• Main findings: Next of kin of 9 730 decedents were approached at the time of registration of death and 55% consented to be approached later and agreed to do a VA interview by a trained field-worker; 83% of physician-reviewed VA data were considered high-quality for determining underlying causes and 22.3% of all the deaths were due to HIV/AIDS, much higher than the proportion reported in the national statistical office.
• Added knowledge: Implementing the VA on a national scale was achievable but significant challenges in recruiting next of kin, emphasising a need to formalise VAs within the country's death notification system.• Global health impact for policy and action: Accurate cause-of-death data are crucial for policymakers to make informed decisions about the country's health system and could be supported by using VAs, particularly for the deaths that occur outside health facilities.

Background
Reliable, continuous, and timely mortality and causeof-death (COD) data are essential for improving health and population policies and supporting countries to respond to emerging health threats and epidemics [1,2].The importance of recording vital events is well recognised, and Civil Registration and Vital Statistics systems (CRVS) that provide continuous data on births, deaths and COD are now seen as a fundamental data source to monitor the Sustainable Development Goals [3].Fifteen of the goals require CRVS data, and 14 of the indicators require cause-specific mortality [3].However, in many low-and middle-income countries, low infrastructure and fiscal investments in civil registration systems have resulted in countries having low levels of death registration and limited or no medical certification of COD.
Verbal autopsy (VA) is a method used to collect and analyse COD information based on an interview conducted with the next of kin or close caregiver about the illness and circumstances leading up to death.Although VA is an imperfect tool to determine the COD, it is often the only population-health option in identifying the COD for out-of-facility deaths [2].It has been suggested that in countries with limited access to health services or medical officers, more extensive use of VA could help fill the information gap [4].In 2017, de Savigny and colleagues [5] presented a detailed systematic view of how VA can be integrated into CRVS.The Bloomberg Philanthropies Data for Health Initiative conducted a structured mapping exercise to identify and map current system responsibilities and data flow for CRVS systems in 16 countries [6].Both studies revealed the challenges involved with integrating VA into CRVS systems.To date, we are not aware of any country that has fully integrated VA into CRVS.
In 2005, the first international technical standards and guidelines for VA were introduced [7].The 2007 VA standard instrument includes separate questionnaires for three age groups with a defined VA list (Annexure 1, Table 2) of causes and corresponding codes from the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) [8].In 2016, questions were added or edited to reach full compatibility with the available automated analysis methods to reduce clinician burden in reviewing questionnaires [9].
South Africa has made great strides in increasing geographic coverage of death registration [10] following integration of the 'homeland' areas and the enactment of the Births and Deaths Registration Act of 1992 [11].Deaths must be registered within 72 h.Recent estimates indicate that completeness for persons above the age of 2 years is over 90% [12].Despite these improvements in death registration, there are still concerns about the quality of data relating to COD.These include a high proportion of deaths with illdefined causes (13%), with an additional 13% having a COD that is not valid as an underlying cause in 2016, under-reporting and misclassification of HIV deaths, and an inaccurate profile of injury deaths [13].The extent of these problems differ at district and national levels [14], and arise from a combination of certifying medical doctors not having access to a full medical history at the time of certification, poor certification practices, and some of the deaths being certified through an affidavit completed by a headman (traditional area administrator) rather than a medical certificate.Although South Africa has three health and demographic surveillance sites [15][16][17] as well as a Child Health and Mortality Prevention Surveillance (CHAMPS) site [18] that routinely use VA to track COD, there has never been national implementation of VA.
The South African National Cause-of-Death Validation Project aims to validate the CRVS COD information by collecting VA, medical, and forensic pathology record data on the COD for a national sample of deaths and then compare this with information recorded in the CRVS system.The initial data analysis of the VAs demonstrated the feasibility of setting up a national collection of VA data [19].This paper presents the methodology and findings related to VAs and highlights important issues for countries considering using VAs within a national initiative.

Methods
We conducted a cross-sectional study that collected data for a fixed-period census of deaths that occurred in a nationally representative sample of health subdistricts in South Africa during parts of 2017 and 2018, using a probability proportional to population size sampling strategy.A detailed methodology is reported elsewhere [19].
In the first phase of data collection, funeral practitioners who serviced the sampled area and were designated by the Department of Home Affairs as official registration agents were recruited to the study.Since the Protection of Personal Information Act (POPIA) [20] of 2013 precludes Department of Home Affairs or the Department of Health from sharing personal information of the next of kin without their consent, it was necessary to work through funeral practitioners and Department of Home Affairs officials as intermediaries to obtain consent to participate in the study.They were asked to share information about the study with the family or informant and requested permission to be contacted by researchers.
Trained fieldworkers contacted consenting informants and arranged VA interviews at least 3 months after the date of death.In addition, the medical records and forensic pathology records were collected from facilities serving the selected areas.The VAs and records were reviewed by medical doctors trained in medical certification of COD.In the final phase of the study, the underlying COD reported in the CRVS system will be validated against the underlying cause identified through the highest level of evidence collected in the study for each decedent.

Sampling and sample size determination
The study population comprised registered deaths from 1 September 2017 until 13 April 2018.We randomly selected 27 sub-districts (Figure 1) for inclusion of all the registered deaths that occurred over a 3-month period in these areas.This accounted for 10.5% of the total population with a median size of 116 000 population per selected sub-district.Using pseudo stratification, three sub-districts were selected from each province according to socio-economic status based on the population size.Due to the true frequency of specific causes of death, the error rate and the extent of clustering that would arise through geographic correlations, sample size determination was not feasible.Instead, scenarios were considered based on the estimated COD profiles, considering the correction factor to be 50%, allowing for a design effect of 2 and a response rate of 85%.It was determined that a sample size of 13 000 deaths would produce a 2-3% precision on the correction factor for HIV/AIDS; 4-5% for cerebrovascular disease; and 7% for diabetes mellitus and interpersonal violence.
During the first 6 weeks, we realised that recruitment of next of kin was extremely challenging.The study period was therefore extended to 8.5 months after which 9 730 informants were approached, with 65% consented.The study protocol was amended to increase the sample size by including deaths of decedents who died in a health facility or who were referred to forensic pathology services.

Data collection
We modified the WHO 2016 questionnaire to start with the narrative which outlines the events leading up to the death in order to establish rapport during the interview and prevent the respondent repeating the story of events after having provided these details through the questions.Minor changes were made to clarify questions related to maternal deaths and to injuries.We used the Open Data Kit (ODK) dictionary to set up the questionnaire in KoBotoolBox, an online data collection tool [21] with translations into eight local languages.A hard copy of the information sheet was given to the respondents to sign and keep.
The KoBotoolBox data collection form made provision for digital signatures.
A team of 84 fieldworkers, selected on the basis of having a university degree or a school-leaving certificate with adequate fieldwork experience, were trained in conducting VAs.Each fieldworker received a tablet containing the data collection tools and fieldworker manual.The fieldworkers were also trained to capture photographic images of de-identified medical and forensic records [22].During the training, continuous assessments identified issues that required further input for the trainees.

Identification of cause of death
A total of 51 physicians with clinical experience in the South African Public Health Service reviewed the collected information and identified the underlying COD.The physicians were orientated to the VA instrument, how to review completed VA questionnaires and use of the customised data collection tools.In order to standardise COD information, physicians were trained in the principles of ICD-10 medical certification of COD [8] and were given standard operating procedures for record reviews.
Anonymised responses to the VA interviews were batched into 40 and independently reviewed by two physician reviewers.Once the batch was completed, quality assurance reviewers evaluated the submissions and identified cases where reviewers assigned different underlying COD.In these cases, re-examination and discussion were necessary to reach consensus.
Physician reviewers also assessed the quality of the clinical information available to identify the COD.They used a subjective score on a scale of 1 (very poor) to 5 (excellent) based on the consistency and coherence of clinical information provided in the narratives and the questionnaire responses.In addition, they scored the sufficiency of the information, on a scale of 1 (very poor) to 5 (excellent), to make a decision about the underlying COD.

Data processing and analysis
We used IRIS V5.8.1 automated software [23] to code the text from the medical certificates and select the underlying COD in ICD-10.We extended the dictionary of medical terms that was developed for IRIS for the Western Cape local mortality surveillance system [24] by adding terms that were commonly used by physicians.Two researchers trained in ICD-10 coding and a co-principal investigator manually coded records that were rejected by IRIS.In addition to the physician reviewers' assessment of the information, the quality of the physician reviewed COD data was assessed using ANACONDA software [25].It checks for biological implausible causes of death based on age and sex.Furthermore, it identifies unusable codes (impossible as underlying causes of death, intermediate causes of death, immediate causes of death, insufficiently specified causes, ill-defined symptoms) for the underlying cause of death.
The cause-specific mortality fractions (CSMF) based on the VA data reviewed by physicians was compared with that from the COD data reported by Statistics South Africa Stats (Stats SA) [26] based on the registered deaths (death notification forms including the medical certificate of cause of death).Stats SA code the information written on the medical certificate of cause of death to ICD-10 and uses the Automated Classification of Medical Entities (ACME 2011) in addition to IRIS to identify the underlying cause of death [26].The COD data from both sources were aggregated to the South African National Burden of Disease (SA NBD) list [27] (shown in Annexure 1) which has been developed to suit the characteristics of the country's disease profile and the quality of routine data.
The CSMF derived from the physician reviewed VAs were compared with those derived from the InterVA-5 algorithm [28] according to the VA list of conditions [9].In this article, the VA results differ slightly from the preliminary findings, because some changes were made to the VA data through additional data cleaning.In addition, we have calculated the InterVA-5 CSMF by aggregating the likelihoods of causes (for up to three conditions for each case) according to the InterVA-5 guidelines compared with the profile provided in the previous report [19] which was based on the frequency of the most likely cause for each case.Approximate 95% confidence intervals (CI) have been calculated assuming the proportions follow a binomial distribution.

Ethical consideration and permissions
Ethical clearance for research involving human participants was obtained from the SAMRC Ethics Committee, and the Health Research Ethics Committees at provincial health facilities.This project was reviewed in accordance with Centers of Disease Control and Prevention (CDC) human research protection procedures and was determined to be research, but CDC investigators did not interact with human subjects or have access to identifiable data or specimens for research purposes.
Funeral practitioners and Home Affairs officials identified all the deaths that occurred in the sample area within the study period.They recruited the next of kin, documented the contact details of consenting informants, and explained that researchers would contact them to arrange the VA interview.
During the fieldworker training the importance of confidentiality was explained.Project staff signed a confidentiality agreement from the undertaking to work ethically and ensure confidentiality.Personal information of the decedent was de-identified and a unique study ID allocated [19].The study ID was applied to the individual patient records in multiple formats and anonymised datasets were created.

Results
A total of 353 funeral practitioners and 95 Home Affairs offices were engaged to recruit next of kin (Table 1).Neither the next of kin nor their dwelling could be located for 560 decedents (8.9%).Out of the 5 756 located dwellings, there was a high response rate and 85.2% completed an interview resulting in an overall response rate of 55%.The main reasons for refusing an interview are shown in Table 2.
After completion of data collection, it was established that 36 976 deaths were registered during the study period and the VAs accounted for 15% of the target sample sub-districts.The geographic location where the interviews were conducted was often outside the sampled sub-districts, as the example of KwaZulu-Natal province in Figure 2 shows.The proportion of VAs falling within the boundaries of the designated sampled sub-district varied by subdistrict.For instance, Jozini sub-district had a very good response rate with the majority of addresses falling within the target area.In contrast, the southern sub-district of Richmond had as many VAs falling outside of the designated area as compared to those which fell within the designated area.
The majority of VAs were conducted within the recommended time interval since the death occurred (83.0%), with a median time between death and the VA interview of 9.2 months.However, a small proportion (0.2%) were conducted less than 3 months since the death occurred, and 16.7% were conducted more than 12 months since the death occurred (median interval of 13.6 months).
Table 3 shows an assessment of the quality of VA information.The information was subjectively assessed on a scale of 1 = (poor) to 5 = (excellent) based on the clinical consistency of information in the narrative and the questionnaire responses, as well as the sufficiency of the VA information for purposes of certification of COD.The physicians scored the majority (81.6%) of the VAs as good quality (score 3-5), while less than 13.2% of the records were assessed as poor-quality information (Table 3).About 66% of VAs had sufficient information to assign the underlying COD (score 3-5).Out of the 5 086 cases with complete information on both criteria, 61.3% had exactly the same score for quality and sufficiency.The kappa statistic was 0.45 (95% confidence interval (CI): 0.44-0.46),indicating a moderate level of agreement between the two dimensions.
The two independent physician reviewers selected the same underlying cause of death in 56.9% of the VAs, while consensus was reached after initial disagreement for 31.7%, while 10.0% required a panel decision to reach consensus, the remaining 1.3% were flagged for review by a maternal mortality specialist to exclude maternal deaths.In 68% of the deaths, the cause of death was coded to a usable code.Overall, 8.7% of the underlying COD were coded to illdefined signs and symptoms, and 16.6% of the causes were considered to have insufficient specification within an ICD chapter.
Figure 3 shows the leading causes of death from the physician reviews compared with vital statistics using the SA NBD list (Annexure 1, Table 1) [27].A slightly higher proportion of the VAs (15.7%) were coded to ill-defined natural causes as compared to 13.3% of deaths in the Stats SA data.However, the ranking and proportions of the specified causes differed considerably.The VA identified 22.3% HIV/ AIDS and 6.9% TB deaths, whereas in Stats SA data the HIV/AIDS and TB deaths accounted for 4.9% and 6.7%, respectively (Figure 3).According to WHO ICD-10 coding guidelines, TB deaths with comorbid HIV are allocated to HIV/AIDS as the underlying cause, while TB deaths without co-morbid HIV are coded to TB as the underlying cause-of death (UCOD).Compared with the SA NBD study [29] albeit 5 years earlier, the VA profile appears more realistic than the Stats SA data, which are known to have extensive misattribution of HIV as a cause of death [30].
An additional feature of the VA data has been the use of 4-digit ICD-10 codes.This makes it possible to identify the deaths with underlying HIV disease that resulted in TB (B20.0)1 which cannot be differentiated in the Stats SA data using the 3-digit ICD-10 codes.Almost half (49.3%) of the HIV/AIDS deaths (604/1 224) had associated tuberculosis, while 61.3% of the TB deaths had underlying HIV (604/984).These are important metrics for monitoring HIV and TB programmes.
From Figure 3, we also observe that the ranking of leading causes of non-communicable diseases differ from Stats SA.The Stats SA data rank diabetes highest followed by stroke and hypertensive heart disease, whereas it is the opposite for the VA data.The VA identified a slightly higher proportion of deaths from external causes than Stats SA (12.8% vs 11.2%).
Figure 4 shows different profiles of injury deaths.Homicide accounted for 35.3% of the VA deaths and only 15.0% of Stats SA causes.This appears to be related to the high proportion of ill-defined  unintentional deaths (48.1%) in the Stats SA data shown in Figure 4.The ranking of the external causes of death based on the physician reviewed VAs aligns more closely with the SA NBD study [29] and the 2017 Injury Mortality Survey [31] which rank homicide highest, followed by transport and then suicide.CSMF derived by InterVA-5 and ranked to the top 15 causes from the VA list is compared with those from the physician coded VA in Figure 5.Although there is considerable similarity in the selection of the underlying COD, physicians assigned more cases to indeterminate underlying COD than the InterVA-5 tool, and there are some notable differences in the CSMF.Physicians identified lower proportions of acute cardiac disease and digestive neoplasms, and higher proportions of HIV/AIDS-related deaths, other unspecified non-communicable diseases, and other unspecified neoplasms compared with InterVA.

Discussion
The study demonstrates that collecting COD data on a national scale using VA is achievable and provided good-quality COD information.The participation rate of next of kin was good once they had been identified and located, with 93.4% completed an interview.The physician reviewers found goodquality COD information was provided in 83% of the VAs.Of the identified COD, only 8.7% were coded to ill-defined natural causes and 68% were assigned a specific and valid underlying cause of death.The remaining 16.6% were assigned an underlying cause of death without sufficient specification.Although the sample cannot be considered nationally representative when compared with SA NBD estimates [29], the VAs provided a more realistic proportion of HIV/AIDS deaths and better information related to external causes of death than COD data from the CRVS system.Improved information related to injuries was obtained from the additional information provided by the narratives compared to Stats SA data.
The use of experienced interviewers was beneficial as it resulted in a very low refusal rate.Our experience of selecting fieldworkers was positive and is aligned with the criteria mentioned in the WHO manual for training of verbal autopsy interviewers.The educational level of VA interviewers varies across VA studies, however the impact of the educational background of VA interviewers on the quality of the VA information collected has not been formally investigated [32].Some authors claim that accurate information can be collected by welltrained lay people from the community, whilst others advocate for medically trained interviewers [32,33].Our choice of VA interviewers was influenced by the experience in HDSS in South Africa, where successful VA interviews have been conducted by lay people [17].
Recruiting the next of kin through funeral parlours and Home Affairs offices proved only partially effective due to the challenges posed by the Protection of Personal Information Act (POPIA) [20].Under POPIA, the Departments of Home Affairs and Health cannot share next of kin information without consent.As a result, intermediaries, such as funeral practitioners and Home Affairs officials, were used to obtain consent.For national implementation of VA, a regulated process will be needed to identify deaths occurring outside of health facilities and to initiate contact with the next of kin.
The use of both physician review and an automated computerised model to ascertain probable COD provided similar results.However, physician review is time-consuming and costly.Variability in the identification of the underlying COD between physician reviewers further necessitates a review panel.Although automated computerised models such as InterVA are cheaper, faster, more consistent, and can be considered for routine coding of high volumes of VAs, quality assurance processes will still be important.Byass et al. [34] compared physician coded VA with InterVA-4 assigned COD from some African and Asian countries, and found strong concordance between physician coded VA and InterVA-4 assigned COD, however, they could not prove which approach provided the true cause of death [34].
Whilst the VA narrative is not used by the algorithms to assign a COD, clinicians find this information critical in determining the COD.In addition, the narrative provided an opportunity for the interviewer to establish rapport with the respondent, thus we feel strongly that the narrative be conducted at the beginning of the interview.

Limitations
The COD patterns presented in this study cannot be assumed to be nationally representative because of the low sample realisation of (55%) VAs achieved.However, the sample does have national coverage, and the results are largely consistent with the national burden of disease profile.The potential bias due to poor sample realisation is not expected to have a major impact on the estimation of correction factors, which was the main objective of the project, however the extent and nature of potential biases remain unknown.

Conclusions
Despite challenges in recruiting the next of kin, our study has demonstrated the feasibility and community acceptability of conducting VAs to ascertain improved COD information.VA could be used at a national scale providing the recruitment of next of kin can be institutionalised into the routine processes for registration of death.It is expected that the use of VA will contribute information particularly for the deaths that occur outside health facilities.We recommend that specific questions in the 2016 WHO VA be clarified to make it easier for interviewers.Reorganising narratives to be conducted at the beginning of the interview worked well as a way of engaging the respondent and orientating the fieldworker.

Figure 1 .
Figure 1.Map of selected health sub-districts and provincial boundaries, South African national cause of death validation project 2017/18.

Figure 2 .
Figure 2. Map of KwaZulu natal showing selected health sub-districts and the address of verbal autopsy interviews conducted for South African national cause of death validation project 2017/18.

Figure 3 .
Figure 3.Leading causes of death based on 2017 stats SA data compared with physician reviewed verbal autopsy data aggregated to National Burden of disease list.

Figure 4 .
Figure 4. Leading injury-related causes of death stats SA, 2017 and physician reviewed verbal autopsy data aggregated to national burden of disease list.

Figure 5 .
Figure 5.Comparison of verbal autopsy cause-specific mortality fractions based on physician reviews with InterVA-5 for the South African national cause of death validation project 2017/18.

Table 1 .
Total number of funeral parlours, department of home affairs offices and next of kin recruited for the South African national cause of death validation project 2017/18 across the three subdistricts in each province.

Table 2 .
Response category and reason for refusal to participate in verbal autopsy interview (N = 5 756), South African national cause of death validation project 2017/18.

Table 3 .
Physicians' assessment of quality and sufficiency of information from verbal autopsy (N = 5,375) for the South African national cause of death validation project 2017/18.

Table A2
provides the ICD-10 codes for the 64 categories in the VA list.

Table A2 .
ICD-10 codes for each category of the VA list.